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ABSTRACT 

Meta analytic procedures recommended by various 
authorities were the subject of a literature review designed not to 
discuss the relative merits of contrasting recommendations, but to 
find what is actually in the literature. The sample reviewed included 
89 articles published between 1986 and 1992, from 2 journals and 2 
information databases. Meta analyses were coded for a number of 
variables. Most reported the databases used to find the studies. The 
median number of studies synthesized by the data analysis was 48. 
About three-quarters of these reported collecting and aggregating 
mean differences. Of the 66 that examined mean differences, 55 
calculated and reported these differences as standardized mean 
differences. Fifteen studies reported effect size, eight used the 
standard normal deviate, and eight used some other method. A large 
variety of statistical methods was reported for the analysis of the 
relationship of moderator variables with effect size. Fort^ -seven 
studies reported an overall test of homogeneity of effect sizes. One 
implication of the study for researchers is that, given the diversity 
of approaches to meta analysis, a good part of the potential audience 
may well prefer a meta analytic approach that differs from that 
chosen by the researcher. Five tables present information on trends 
in meta analysis. An appendix lists studies that appeared in the two 
main journals reviewed. (SLD) 
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TRENDS IN PUBLISHED META-ANALYSES 
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The prominence of meta-analyses in the research literature 
in education, psychology, and other fields has become evident. 
Unless some unexpected change occurs, investigators and other 
students in almost all fields can expect to encounter meta- 
analytic methods, either as a producer or as a consumer of the 
research literature, for some time to come. 

At least two sources of confusion currently complicate the 
study of meta-analytic methods. One is the differing conclusions 
reached by separate meta-analyses on the same or similar topic. 
The other is the vari. -ion in meta-analytic procedures which have 
been recommended by various authorities. The latter concern led 
to the study reported here. 

Differences in recommended procedures concern increasingly 
varied aspects of meta-analysis. The most obvious variation is 
in ho.w to compute and report effect size, differences which even 
if mathematically equivalent can complicate the task of 
understanding the literature. Other variations concern 
procedures for analysis of moderator variables (even whether to 
code moderator variables) , the application of tests of 
homogeneity to the distribution of effect sizes, limitations on 
the nuirtber of effect sizes to come from a single study, the 
importance of including unpublished literature, if and what 
corrections to make of the effect size data, and whether to 
include a rating of methodological quality of individual studies 
as a moderator variable. 

The present study was not designed to discuss the relative 
merits of contrasting recommendations but was intended to find 
out what is actually in the recent literature. Results of such a 
literature survey might possibly help one to decide who to 
believe if not who one should believe. 



METHOD 

Selecting Studies: The sample examined in this study 
consisted of 89 articles identified as meta-analyses and 
published between 1986 and 1992. They were taken from three 
sources, two of which are presumptive major sources of meta- 
analyses for psychologists and educators: (1) All articles 
identified as meta-analyses in the Review of Educational Research 
(RER) from 1986 through the first three issues of 1992, 18 in 
all; (2) all accessible articles identified as meta-analyses in 
the issues of the Psychological Bulletin (PB) from 1986 through 
1991 (three issues missing during the search period) , a total of 
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43; and (3) a group of 28 meta-analyses 1986-92 sampled from ERIC 
and PsychLIT and not published in either the RER or PB. 



Coding Studies The following information was recorded for 
each study: 

publication source 
year of publication 

reference to previous meta-analysis on same topic 

if inclusion criteria were described 

data bases used 

number of studies synthesized 

nature of statistics aggregated and synthesized 
corrections, if any, made to individual effect sizes 
effect size formula used 

if individual studies were rated for quality 
presence of statistical significance tests for 

moderator variables 
statistical tests used for moderator variable 

analysis 

Except for number of studies, all variables were coded as 
present or absent. Accordingly, some of the above variables were 
broken down into sub-categories, e.g. , t-tests for the last item 
above (Table I) . 

Data Analysis: An appropriate procedure for analyzing the 
data beyond simple description was not obvious. Accordingly, the 
basic procedure was simply to tabulate the data and note the 
frequencies for each coded study characteristic. Cross- 
tabulating most of the variables with year of publication and 
publication source was a supplemental procedure. 



RESULTS 

The frequencies of all of the coded study features are 
presented in Table I. The distribution of the 89 sample meta- 
analyses by year and by publication source can be found in Table 
II. The Appendix contains the bibliographic citations of all 
the met: analyses and how each was coded (last two pages of the 
Appendix) . 

Searching and Selecting Studies: Most meta-analyses in the 
study sample reported the data bases used for finding studies. 
The frequencies of reported use were as follows: 

ERIC 34 

PsychAbs/PsychLit/PsychlNFO 50 

Dissertation Abstracts 3 0 

Medline 7 

Other Computer 18 

Most studies indicated that use of the above was supplemented by 
manual search and in some cases by personal corresondence or 
contact. 



ERLC 



Seventy- three (82%) studies reported inclusion (or 
sometimes exclusion) criteria for the selection Gf studies. 
Usmally these were specified quite clearly, but there were a few 
cases tough to call. 

Number of Studies: The median number of studies synthesized 
by the meta-analyses in the sample was 48. The numbers ranged 
from six to 411. The latter was an outlier, for the next highest 
number was 172. 

Moderator Variables: The coding of individual studies for 
potential moderator variables has normally been considered as an 
essential part of the meta-analytic process. This was done in 83 
(93%) of the meta-analyses in the sample. Some of the exceptions 
were studies for which classification as a meta-analysis, rather 
than say a secondary data reanalysis, was questionable. 

One of the noteworthy features of the original Smith and 
Glass meta-analysis (Glass, 1976) was to rate the quality of the 
individual studies and use that rating as a moderator variable 
rather than a basis for excluding studies. This practice was 
followed in only sixteen (18%) of the meta-analyses in the 
sample, and there was no statistically significant association of 
this practice with year of publication or publication source 
(Table III) . 

T ypes of Statistics Collected: Of the sample of 89 meta- 
analyses, about three-quarters (66) reported collecting and 
aggregating mean differences. Correlations were collected in 
sixteen (19%) of the studies. Twenty-four (27%) of the studies 
reported other statistics, such as proportions and in one instance 
variances, for a major part their syntheses. These numbers total 
more than 89 since some studies reported amassing more than one 
type of statistic. 

There was a question whether a few of the studies in the 
sample should be considered as secondary data reanalyses and not 
meta-analytic. These were studies which synthesized raw data, 
usually from test-norming reports. Nonetheless, they were 
included and put in the "other" category indicated above. 

Method of Calculating Effect Size: Of the sixty-six studies 
which examined mean differences, by far the large majority (55) 
calculated and reported mean differences as standardized mean 
differences, that is, the raw score difference between group 
means divided by a standard deviation of individual scores. 
Fifteen studies reported effect size as a correlation, eight used Z 
(standard normal deviate) , and eight used some other method. 

What to use as a denominator when computing standardized 
mean differences became an issue not long after Smith and Glass 
officially launched meta-analysis and proposed the use of the 
standard deviation of the control group. Whether because of the 
persuasiveness of the argument to use the pooled standard 



deviation, because of the difficulty of finding and extracting the 
control group standard deviation, because of the editorial policy 
of the Psychological pulletin which had the largest number of 
studies in the sample, or for some other reason, the majority of 
studies did not use the control group standard deviation. Of the 
twenty-two which did, only six were published after 1988, a 
significant drop-off (chi-square - 9,92, p <.01). 

No other trends in the data about the various methods of 
reporting effect size were noted. 

Statistical Analysis o£ Moderator Variables; A large variety 
of statistical methods were reported for the analysis of the 
relationship of moderator variables with effect size. Various 
studies reported more than one method, sometimes necessitated by 
the differences in the type of statistics collected and 
aggregated. Tallies of the various methods were as follows: 



t tests 12 

Analysis of variance (or an analog) 18 

Significance of single correlations 11 

Multiple regression 12 
Successive homogeneity tests of subsets 21 

Other 11 

No significance test 9 



The uses of the various tests were distributed fairly evenly 
over the surveyed years (Table IV) , but it was noted that ANOVA 
procedures were more often used during the first part of the period 
while simple correlation and multiple regression were more frequent 
during the latter part. 

In some cases where no statistical test of moderator 
variables was reported, the decision was deliberate because an 
overall test of homogeneity of effect sizes indicated no 
significant heterogeneity. (The overall test for homogeneity was 
somewhat arbitrarily counted as not a moderator variable test.) 

Tests of Homogeneity : Forty-seven (52%) of the studies 
reported an overall test of homogeneity of effect sizes to see if 
there were significant differences among them which might be 
linked to moderator variables. The RER meta-analyses had a 
smaller proportion of these studies (six of eighteen or 33%) , but 
this pattern was not statistically significant. However, there was 
a significant increase in the use of this test after 1988 (Table V) . 



DISCUSSION 

Suggestions for further study, if there is merit in surveys 

of the current research literature, can just as well start with a 
Monday-morning quarterbacking attitude. With the benefit of 

hindsight, what would be done differently if this study were to 

begin now? The following ideas have come to mind, in no 
particular order: 
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1. Tally the number of effect sizes reported by 
each meta-analysis in addition to the number of 
studies included. 

2. Expand the number of studies sampled from 
journals other than the RER and the PB. 

3. Draw studies from other databases, at least 
MEDLINE, to check if trends and tendencies are 
different. With this also find a library which has 
most of the journals covered by MEDLINE (a real 
stumbling block; the idea of using MEDLINE was 
considered but then abandoned for this study) • 

4. Code each meta-analysis according to whether it 
included unpublished studies. 

5. Code each meta-analysis according to indication 
of what source served as major reference point for 
meta-analysis methodology, whether Glass et al . , 
Hedges and Olkin, Hunter and Schmidt, Rosenthal, or 
someone else. 

6. Code for any indication of testing a priori 
hypotheses. 

7. Pay more attention to the reported conclusions and 
try to rate them on substantive significance. Then try 
to relate the ratings to other characteristics, such as 
major point of reference (above) . 

8. For reasons not limited to an attempt to increase 
reliability of coding, make this something other than 
a solo effort. What are graduate students for 
anyway? 

As well as figuring out what might have been done 
differently, there can be speculation about a more ambitious 
study. Assuming a more global domain of meta-analyses has been 
determined and clarified and kinks in the sampling procedure can 
be worked out, a quantitative synthesis of meta-analyses could be 
attempted (let us hope and pray, probably futilely, that the term 
"meta-meta-analysis" would be avoided) . Collected as dependent 
variables would be overall effect sizes as well as number of 
significant associations of moderator variables with effect 
sizes. Coded study features, such as those considered for this 
study, would serve as moderator variables for this overall 
synthesis. 

Such a study could shed light on whether the Hunter-Schmidt 
procedures do in fact lead to an appreciable smaller variance of 
effect sizes and fewer significant associations of moderator 
variables with effect sizes than the Hedges-Olkin procedures, as 
implied by Raudenbush (1991) . Finding a difference, however, would 
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still not indicate which approach is more valid. 



Reportin g of Meta-Analyses: As far as reporting meta- 
analyses is concerned, this study has few implications, certainly 
little beyond the recommendations of Cooper (1989) . And given the 
state of the art of meta-analytic theory and the occurrence of 
conflicting results- from similar meta-analyses, this study in.no 
way supports a call for greater standardization of procedures; we 
need to know what is right before deciding what to standardize. 
However, there is one implication that could help the consumer. 
That is for the authors of reviews to realize that a good part of 
the potential audience is not of the same meta-analytic camp as 
they are. One who is in the Hunter-Schmidt camp, for instance , 
should try to explain as clearly and simply as possible what it 
means to correct for artifacts and why and maybe also why an 
overall test of effect size homogeneity was not used. For the 
consumer, this would not make the meta-analytic fields any 
smaller but it could make traversing them smoother. 
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TABLE I 

TALLIES OF CODED STUDY FEATURES 



CODE STUDY FEATURE 

a Published in 1986 

b Published in 1987 

c Published in 1988 

d Published in 1989 

e Published in 1990 

f Published in 1991 

g Published in 1992 

h Reference to previous meta-analysis, similar topic 

j Inclusion (or exclusion) criteria described 

k ERIC used 

I PsychAbs/Psychlnfo/PsychLit Used 
m Dissertation Abstracts used 
o MedLine used 

n Other computer data base used 

q Number of studies synthesized 
r ' Mean differences aggregated 

s Correlations aggregated 

t Other statistics aggregated 

u Effect size corrected for sample size 

v Effect size corrected for attenuation 

w Effect size corrected for other artifacts 

y No corrections of effect size reported 

z Moderator variables coded 

aa Stand, mean diff., any standard deviation 

bb Stand, mean diff., control group standard deviation 22 

cc Effect size reported as correlation 

dd Effect size reported as Z 

ee Other mode for reporting effect size 

ff Means not the statistic aggregated 

gg Test of homogeneity for moderator variables 

hh t test used for moderator variable analysis 

jj ANOVA used for moderator variable analysis 

kk significance of separate correlations tested 

II multiple regression used for moderator analysis 
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TABLE H 



STUDIES IN SAMPLE BY PUBLICATION AND YEAR 





1986 


1987 


1988 


1989 


1990 


1991/2 




Psych Bull 


4 


7 


8 


5 


10 


9 


43 


Rev of Ed Res 


1 


2 


2 


5 


3 


3 


18 


Other 


5 


6 


2 


5 


4 


6 


28 




12 


15 


12 


15 


<7 







TABLE III 

NUMBER OF META-ANALYSES WH CH RATED QUALITY OF INDIVIDUAL STUDIES 

1Q86 1987 1988 1989 1990 1991/2 

Studies which Rated Quality 4 3 0 4 1 4 

Total Number of Studies 12 13 12 15 17 18 
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TABLE W 

STATISTICAL TESTS USED FOR MODERATOR VARIABLE ANALYSIS 





198$ 


1987 


1988 


1989 


199C 


1991/2 


TOTAL 


t test 
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2 


1 
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12 


ANOVA 
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18 


Correlation 


2 


1 
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11 


Multiple Regression 


1 
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1 


12 


Successive Homogeneity 


2 
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21 


Other 


1 


3 


2 
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11 


Number of Studies 


12 


15 


12 


15 


17 


18 


89 



ERIC 



TABLE V 

STUDIES WHICH TESTED FOR HOMOGENEITY OF EFFECT SIZES 

1986-88 1989-9 



Homogeneity Test 14 33 47 

No Homogeneity Test 25 17 42 



39 50 89 



Xl (1 d.f.) = 9 78 P < 01 
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APPENDIX TO TREMDfl IN PUBLISHED META-ANALYSES 

Burton L. Grover AERA, Atlanta, GA. 

Western Washington University April 1993 



Studies from PSYCHOLOGICAL BULLETIN 
1986-1989 

101 Achenbach, Thomas, Stephanie McConaughty, and Catherine 
Howell. "Child/Adolescent Behavioral and Emotional Problems: 
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103 Booth-Kewley Stephanie and Howard Friedman. "Psychological 
Predictors of Heart Disease: A Quantitative Review," 
Psychological Bulletin , Vol. 101, No. 3, May 1987, pp. 343- 
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106 Bushman, Brad and Harris Cooper. "Effects of Alcohol and Human 
Aggression: An Integrative Research Review, " Psychological 
Bulletin , Vcl. 107, No. 3, May 1990, pp. 341-354. 

107 Carlson, Michael and Norman Miller. "Explanation of the 
Relation Between Negative Mood and Helping," Psychological 
Bulletin . Vol. 102, No. 1, July 1987, pp. 91-108. 

108 Cohn, Lawrence. Sex Differences in the Course of Personality 
Development: A Meta Analysis," Psychological Bulletin , Vol. 
109, No. 2, March 1991, pp. 252-266. 

109 Durlak, Joseph; Teresa Fuhrman; and Claudia Lampman. 
"Effectiveness of Cognitive-Behavior Therapy for Maladapting 
Children: A Meta-Analysis, " Psychological Bulletin , Vol.110, 
No. 2, September 1991, pp. 204-14. 

110 Dush, David, Michael Hirt, and Harold Schroeder. "Self- 
Statement Modification in the Treatment of Child Behavior 
Disorders: A Meta-Analysis, " Psychological Bulletin , Vol. 106, 
No. 1, July 1989, pp. 97-108. 
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111 Eagley, Alice; Richard Ashmore; Mona Makhijani; and Laura 
Longo. "What Is Beautiful is Good, But ... : A Meta-Analytic 
Review of Research on the Physical Attractiveness Stereotype," 
Psychological Bulletin . Vol. 110, No. 1, September 1991, pp 
109-128. 

112 Eagley, Alice and Blair Johnson. "Gender and Leadership Style: 
A Meta-Analysis," Ps ycholog ical Bulletin. Vol. 108, No. 2, 
September 1990, pp. 223-256. 

113 Eaton, Warren, and Leslie Reid Enns. "Sex Differences in Human 
Motor Activity Level," Psychological Bulletin . Vol. 100, No. 
1, July 1986, pp. 19-28. 

114 Feingold, Alan, "Matching for Attractiveness in Romantic 
Partners and Same-Sex Friends: A Meta-Analysis and Theoretical 
Critigue," Psychological Bulletin . Vol. 104, No. 2, September 
1988, pp. 226-235. 

115 Ford, Kevin; Kurt Kraiger and Susan Schechtman. "Study of Race 
Effects in Objective Indices and Subjective Evaluation of 
Performance: A Meta-Analysis of Performance Criteria." 
Psychological Bulletin . Vol. 99, No. 3, May 1988, pp 330-7. 

116 Hardaway, Richard. "Subliminally Activated Fantasies: Facts 
and Artifacts," Psycholog ical Bulletin. Vol 107, No. 2, March 
1990, pp 177-185. 

117 Hazelbrigg, Mark; Harris Cooper; and Charles Borduin. 
"Evaluating the Effectiveness of Family Therapies: An 
Integrative Review and Analysis." Psychological Bulletin. Vol. 
101, No. 3, 1987, pp 428-422. 

118 Hulin, Charles; Rebecca Henry, and Sharon Noon. "Adding a 
Dimension: Time as a Factor in the Generalizability of 
Predictive Relatioships. " Psycholog ical Bulleitn. Vol. 107, 
No. 3, May 1990. pp 328-340. 

119 Hull, Joy and Charles Bond, Jr. "Social and Behavioral 
Conseguences of Alcohol Consumption and Expectancy" A Meta- 
Analysis," Psychological Bulletin . Vol. 99, No. 3, May 1986, 
pp 347-360. 

120 Hyde, Janet; Elizabeth Fennema; and Susan Lamon. "Gender 
Differences in Mathematics Performance: A Meta-Analysis," 
Psychological Bulletin . Vol. 107, No. 2, March 1990, pp 139- 
155. 

121 Hyde, Janet and Marcia Linn. "Gender Differences in Verbal 
Ability: A Meta-Analysis," pgyr>- ogical Bulletin. Vol. 104, 
No. 1, July 1988, pp. 53-68. 
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122 Johnson, Blair and Alice Eagly. "Effects of Development on 
Persuasion: A Meta-Analysis, " Psychological Bulletin . Vol. 
106, No. 2, September 1989, pp. 290-314. 

123 Kaylor, Jeffrey, Daniel King, and Lynda King. "Psychological 
Effects of Military Service in Vietnam: A Meta-Analysis," 
Psychological Bulletin , Vol. 102, No. 2, September 1987, pp. 
257-271. 

143 Lytton, Hugh and David Romney, "Parents' Differential 
Socialization of Boys and Girls: A Meta-Analysis. 
" Psychological Bulletin , Vol. 109, No. 2, March 1991, pp. 267- 
296. 

12 4 Mathieu, John and Dennis Zajac, "A Review and Meta-Analysis to 
the Antecedents, Correlates and Consequences of Organizational 
Commitment." Psychological Bulletin , Vol. 108, No. 2, 
September 1990, pp 171-194. 

125 Matthews, Karen. "Coronary Heart Disease and Type A Behaviors: 
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Performance," Psychological Bulletin . Vol. 102, No. 1, July 
1987, pp. 53-71. e 

141 Wood, Wendy; Nancy Rhodes; and Melanie Whelan. "Sex 
Differences in Positive Weil-Being: A Consideration of 
Emotional Style and Marital Status," Psychological Bu lletin. 
Vol. 106, No.l 2, September 1989, pp. 249-264. 

142 Wood, Wendy; Frank Wong; and Gregory Chachere. "Effects of 
Media Violence on Viewers' Aggression in Unconstrained Social 
Interaction," Psychological Bulletin . Vol 109, No. 3, May 
1991, pp. 371-83. 
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201 Bangert-Downs, Robert L., Chen-Lui Kulik, James Kulik, and 
Mary Teresa Morgan. "The Instructional Effect of Feedback in 
Test-Like Events," Review of Educational Research , Vol. 61, 
No. 2, Summer 1991, pp. 213-238. 

202 Becker, Betsy Jane. "Coaching for the Scholastic Aptitude 
Test: Further Synthesis and Approval, 11 Review of Educational 
Research , Vol. 60, No. 3, Fall 1990, pp. 373-417. 

203 Chapman, James. "Learning Disabled Children f s Self -Concepts, " 
Review of Educational Research , Vol. 58, No. 3, Fall 1988, pp. 
347-371. 

204 Falchikov, Nancy and David Boud. "Student Self -Assessment in 
Higher Education: A Meta-Analysis, " Review of Educational 
Research , Vol. 59, No. 4, Winter 1989, pp. 395-430. 

218 Feingold, Alan. "Sex Differences in Variability in 
Intellectual Abilities: A New Look at an Old Controversy." 
Review of Educational Research . Vol. 26, No. 1, Spirng 1992, 
pp. 61-4 

205 Friedmamn, Lynn. "Mathematics and the Gender Gap: A Meta- 
Analysis of Recent Studies on Sex Differences in Mathematical 
Tasks," Review of Educational Research , Vol. 59, No. 2, Summer 
1989, pp. 185-213. 

206 Fuchs, Douglas and Lynn Fuchs. "Test Procedure Bias: A Meta- 
Analysis of Examiner Familiarity Effects," Review of 
Educational Research , Vol. 56, No. 2, Summer 1986, pp. 243- 
262. 

207 Hamaker, Christiaan. "The Effect of Adjunct Questions on Prose 
Learning," Review of Educational Research , Vol. 56, No. 2, 
Summer 1986, pp. 212-242. 

208 Hoge, Robert and Theodore Coladarci. "Teacher-Based Judgements 
of Academic Achievement: A Review of Literature," Review of 
Educational Research , Vol. 59, No. 3, pp. 297-313. 

209 Kulik, Chen-Lui, James Kulik, and Robert Bangert-Downs. 
"Effectiveness of mastery Learning Programs: A Meta-Analysis," 
Review of Educational Research , Vol. 60, No. 2, Summer 1990, 
pp. 265-299. 

210 Lundeberg, Mary, and Paul Fox. "Do Laboratory Findings on Test 
Expectancy Generalize to Classroom Outcomes," Review of 
Educational Research , Vol. 61, No. 1, Spring 1991, pp. 94-106. 
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211 Ross, John A. "Controlling Variables: A Meta-Analysis of 
Training Studies/ 1 Review of Education Research , Vol. 58 , No. 
4, Winter 1988, pp. 405-437. 

212 Slavin, Robert. "Ability Grouping and Student Achievement in 
Elementary Schools: A Best-Evidence Syntheses, " Review of 
Educational Research , Vol. 57, No. 3, Fall 1987, ppa 293-336. 

213 Slavin, Robert. "Mastery Learning Reconsidered," Review of 
Educational Research , Vol. 57, No. 2, Summer 1987, pp. 175- 
213. 

214 Slavin, Robert. ."Achievement Effect of Ability Grouping in 
Secondary Schools : A Best Evidence Syntheses , " Review of 
Educational Research , Vol. 60, No. 3, Fall 1990, pp. 471-499. 

215 Stahl, Steven A. and Marilyn Fairbanks. "The Effects of 
Vocabulary Instruction: A Model-Based Meta-Analysis, " Review 
of Educational Research , Vol. 56, No. 1, 1986, pp. 72-110. 

216 Stahl, Steven and Patricia Miller. "Whole Language and 
Language Experience Approaches for Beginning Reading: A 
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59, No. 1, Spring 1989, pp. 65-86. 
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REVIEW OF EDU CATIONAL RESEARCH 
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3 01 Adair, John; Donald Sharpe; and Cam-Loi Huynh. "Hawthorne 
Control Procedures in Educational Experiments: A 
Reconsideration of Their Use and Effectiveness." Paper 
presented to the Annual Meeting of the American Educational 
Research Association, Washington, D.C., April 1987. (ERIC No. 
ED 286926) . 

3 02 Christ iansen-Szalanski, Jay and Cynthia Willham. "The 
Hindsight Bias," Organizational Behavior and Human Decision 
Processes . Vol.48, No. 1, February 1991, pp 147-168. 
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Psychology P Vol. 59, No. 5, 1990, pp 981-993. 

307 Fletcher, J.O. "Effectiveness and Cost of Interactive 
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Formative Evaluation: A Meta-Analysis, " Exceptional Children , 
Vol. 53, No. 3, November 1986, pp 199-229. 
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"Anxiety, Depression, and Perception of Early Parenting: A 
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1990, pp 251-277. 
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1986, pp 310-319. 

308 Holweg, Kurt and Howard Markman, "Effectiveness of Behavioral 
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Preventing and Alleviating Marital Distress, 11 Journal of 
Consulting and Clinical Psychology , Vol. 56, No. 3 , 1988, pp 
440-447. 
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Review," Clinical Psychology Review , Vol. 12, No. 2, 1992, pp 
227-255. 
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310 Mullen, Brian et al . "Boundaries Around Group Interaction: A 
Meta-Analysis Integration of the Effects of Group Size," The 
journal of Socia? psycho locrv . Vol 131, No. 2, 1991, pp 271- 
283. 

311 Multon, Karen; Steven Brown; and Robert Lent. "Relation of 
Self-Ef f icacy Beliefs to Academic Outcomes: A Meta-Analytic 
Investigation," . Journal of Counsel ing Psychology. Vol. 38, No. 
1, January 1991, pp 30-38. 

312 Murphy, Kevin, and William Balzer. "Rater Errors and Rating 
Accuracy," Journal of Appli ed Psychology. Vol. 74, No. 4, 
August 1989, pp 619-626. 
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Grouping: A Meta Analysis." Paper presented at the Annual 
Meeting of the American Educational Research Association, San 
Francisco, April 1986. (ERIC No. Ed 258 451). 

314 Perlman, Daniel. "Age Differences in Loneliness: A Meta- 
Analysis." Paper presented to the Convention of the American 
Psychological Association, Boston, August 1990 (ERIC No. ED 
326 767) . 

315 Salzman, Stephanie, "Father Absence, Socioeconomic Status and 
Race; Relations to Children's Cognitive Performance. 11 Paper 
presented at the Annual Keeting of the American Educational 
Research Association, New Orleans, April 1988 (ERIC No. ED 296 
939) . 

316 Salzman, Stephanie. "Meta-Analysis of Studies Investigating 
the Effects of Father Absence on Children's Cognitive 
Performance." Paper presented at the Annual Meeting of the 
American Educational Research Association, Washington, D.C., 
April 1987, (ERIC No. ED 283 883) 

317 Samson, Gordon; Bernadette Strykowski; Thomas Weinstein; and 
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Student Achievement: A Quantitative Synthesis." Journal of 
Educational Research . Vol. 80, No. 5, May-June 1987, pp 290- 
295. 

318 Schueger, James; Karen Zarella; and Annette Hotz. "Factors 
that Influence the Temporal Stability of Personality by 
Questionnaire," Journal of Psychology and Social Psychology, 
Vol. 56, No. 5, May 1989, pp 777-783. 

319 Suls, Jerry and Choi Wan. "Effects of Sensory and Procedural 
Information on Coping with Stressful Medical Procedures and 
Pain: A Meta-Analysis," Journal and Consulting and Clinical 
Psychology . Vol. 57, No. 3, 1989, pp 372-379. 
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320 Svartberg, Martin and Tore Stiles. "Comparative Effects of 
Short-Term Psychodynamic Psychotherapy: A Meta-Analysis, M 
Journal of Consulting and Clinical Psychology P Vol. 59, No. 5, 
October 1991, pp 704-714. 

320 Swartout, David and David Synk. "The Effect of Age, Education 
and Work Experience on General Aptitude Test Battery Validity 
and Test Scores," USES Test Research Report , No. 50, 1987, 
(ERIC No. ED 310 171) 

3 21 Tenenbaum, Gerson and Ellen Goldring. "A Meta-Analysis of the 
Effect of Enhanced Instruction: Cues, Participation, 
Reinforcement and Feedback, and Correctives on Motor Skill 
Learning," Journal of Research and Development of Education , 
Vol. 22, No. 3, Spring 1989, pp 53-64. 

322 Tobler, Nancy. "Meta-Analysis of 143 Adolescent Drug 
Prevention Programs: Quantitative Outcome Results of Programs 
Participants Compared to a Control or Comparison Group," 
Journal of Drug Issues , Vol. 16, No. 4, Fall 1986, pp 537-567. 

326 Van Sickle, Ronald. "A Quantiative Review of Research on 
Instructional Gaming" A Twenty-Year Perspective." Theory and 
Research in Social Education P Vol. 14, No. 3, Summer 1986, pp 
245-264. 

3 23 Wagner, John III and Richard Gooding. "Shared Influences and 
Organizational Behavior: A Meta-Analysis of Situational 
Variables Expected to Moderate Participation-Outcome 
Relationships," Academy of Management Journal r Vol. 30, No. 3, 
September 1987, pp 524-541. 

324 Wellman, Henry; David Cross, and Karen Bortsch. 'Infant Search 
and Objective Permanence: A Meta-Analysis of the A-Not-B 
Error." Monographs of the Society for Research in Child 
Development, Vol. 51, No. 3, Serial No. 214, 1987. 

325 Witters-Churchill, Laurie and Lee Witters. "Hearing-Impaired 
Students 5 Performance on the Piagetian Liquid Horizontality 
Test: An Analysis and Synthesis . 11 Paper presented at the 

i Internatioanl Symposium on Cognition, Education, and Deafness, 

Washington, D.C., July 1989 (ERIC No. 313 845). 
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327 Yammarino, Francis: Steven Skinner, and Terry Childs, 
"Understanding Mail Survey Response Behavior: A Meta- 
Analysis," Public Opinion Quarterly . No. 55, No. 4, Winter 
1991, pp 613-639. 
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0 ABCDEFGHJKLMPQ 
BID AHCDEFGHJKLMON ORSTUVWYZX ABCDEFGHJKLMPQ 

301 01000001000000 4410010000010000010000100 

302 0000010011110112210001001010100010010000 

303 00001000111100 23101100010100000000000.0 

304 10000000111100 2110010101111000010100000 

305 00001000101000 3211011001010100010000100 

306 10000000101000 291000000111000C011100000 

307 00C UOOOlllOOl 2810000011010000010000000 

308 00100000100000 2410000011011100001000000 

309 00000010101000 1500110101000001110000001 

310 0 00 0010110000000600100011000010 00 00000 00 

311 00000100111100 3901010001100000110000100 

312 00010000100001 1001011001000000110001000 

313 10000000100000 5010000011111000001000000 

314 00001000000000 1401000001000000100010000 

315 00100001100000 8210000011011000000100000 

316 010000001000 0013710000011111000000100000 

317 01000001110100 1410000001011010000100000 

318 00010000000000 8901000011000000100001000 

319 00010000101010 2110010101010000010000100 

320 00000100101010 1910000101100100010000010 

320 01000000100 00 014 3100 0001000000010010 0000 

321 00010000000000 1610010001011000010000000 

322 10000000100001 9810000001011000000001000 

323 01000001100000 7010010001000000111000100 

324 01000000101000 3000100011000000100000000 

325 00010001110000 710000010010000000000000 
201 0000010111010004010000011011000001010000 
2 02 0000100111010004 800100011000011110001000 

203 00100000000000 2110000011011000000000001 

204 00010000110111 5711110001111001110000000 

205 00010001011100 7710010001010000010001000 

206 10000000111100 2210000011111000010100000 

207 10000001111000 6100111010000000101000000 

208 00010000111000 1601000011000000100000001 
2 09 0000100111010010810100011011000011101000 

210 00000 100001101107 100 1000101000000000^010 

211 00100000110101 6510010001011111100100000 

212 01000001111100 1410000011111000000000000 

213 01000001110100 1610000011011000000000000 

214 00001001110100 2910000101011000000000000 

215 10000001110000 5210000011011000000000010 

216 00010001110100 6510000011111000011110000 

217 00010000110000 2200110000000001100000000 

101 01000000101000119010100010000001000000 00 

102 0 000010 010100109 2101100 0101000 0 010100000 

103 01000000101010 8700100011000100000000010 

104 0001000010100013 411000011000110000000000 

105 00100001101000 6910000011011000000101000 
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106 00001001101101 3010010001010000010001100 

107 01000000101000 4410000011010000000011010 

108 00000101101100 6511010001010000010100000 

109 00000100100100 6410010001110000010000100 

110 00010001101100 4810010001010000000000001 

111 00000101111111 6910010001010000010100000 

112 0000100010110114410010001010000010110000 

113 10000000101000 9010010001010000010010000 

114 00100000011100 1801010001000110000000110 

115 10000001100000 5310010001000100010000100 

116 00001000101100 5610011001010000010000001 

117 01000000111000 2010110001010000010000100 

118 00001001100000 4100101101000001110000010 

119 10000000100000 3410000011010000010000100 

120 0000100111100010010010001010000010001100 

121 0010000011100016510000101010000010000100 

122 00010000111101 3810010001010000010000000 

123 01000000101100 6710000001110000001101000 

124 00001000101001124 01001001000000111000100 

125 00100001100000 1600110001000010010000000 

126 00001000101000103 01010001000000100000100 

127 00100000011000 2910110001000100010000100 

128 0000010110000C 3910110101000100010010000 

129 00000100001011 3010110001000101000000001 

130 00001001111001 5810100011110000010000100 

131 0010000110000041101110001000001100000100 

132 10000001110100 1010000011011000000010000 

133 00100000100000 1401000101000000110000010 

134 00001000100000 9310010001010000001010010 

135 00001001101000 5810010001010000001000000 

136 00000100101010 5210010001000100010001000 

137 00100000000000 4300100101000100000000001 

138 01000000001001 3510010001000010000000110 

139 00010000101000123110 00101110000010000100 

140 01000000001000 5210110001010000010100000 

141 00010001101001 9311100110100000010010000 

142 00000101111100 2310010001010000010100000 

143 0000010100110017210010001110000010000010 
218 00000011000000 800100011000000100000001 

326 10000001010000 2 210000011011000000000001 

327 0000010111100111500110101000100110000100 

One line per study. The first three numbers indicate the 
identification number of the study as listed in the bibliography. 

Meanings of code letters for variables may be found 
on Table I of the paper. 
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